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I.  Introduction 

"^Ceapon  systems  are  becoming  increasingly  automated  and  arguably 
some  autonomous  military  systems  have  been  deployed  for  years.  Recent 
advances  in  automated  systems  and  the  possibilities  they  portend  have  gen¬ 
erated  interest  and  anxiety  within  some  militaries  and  defense  ministries, 
and  a  movement  of  non-governmental  activists  seeking  to  ban  fully  auton¬ 
omous  weapons.  In  May  2014,  the  High  Contracting  Parties  of  the  UN 
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Convention  on  Certain  Conventional  Weapons  (CCW)  convened  an  exten¬ 
sive  discussion  of  the  legal  and  ethical  issues  that  autonomous  weapons 
raise,  while  recognizing  that  many  of  these  problems  lie  at  an  uncertain 
point  in  the  future. 

It  is  important  that  normative  development  regarding  autonomous 
weapon  systems  head  down  a  path  that  is  coherent  and  practical.  By  “au¬ 
tonomous  weapon  systems,”  we  mean  systems  “that,  once  activated,  can 
select  and  engage  targets  without  further  intervention  by  a  human  opera¬ 
tor.”  We  draw  this  definition  from  a  2012  U.S.  Department  of  Defense 
policy  directive,  which  remains  the  most  extensive  public  pronouncement 
by  any  State  on  how  it  intends  to  proceed  with  regard  to  research,  devel¬ 
opment  and  deployment  of  autonomous  weapon  systems." 

This  paper  addresses  several  questions  that  are  critical  to  charting  such 
a  path.  First,  are  autonomous  weapon  systems  different  from  other  new 
weapon  systems,  and,  if  so,  how?  Second,  to  the  extent  they  are  different, 
can  and  should  autonomous  weapon  systems  be  regulated  within  the 
framework  of  the  existing  law  of  armed  conflict?  If  yes,  how  should  States 
go  about  doing  so?  If  not,  what  alternative  regulatory  approach  is  appro¬ 
priate? 

We  conclude  that  autonomous  weapon  systems  have  special  features 
that  pose  risks  and  that  create  challenges  in  applying  the  existing  law  of 
armed  conflict.  Nevertheless,  we  conclude  it  is  possible  to  adapt  the  exist¬ 
ing  framework  to  account  for  the  features  of  autonomous  weapons,  and 
that  the  suggested  alternative  of  prohibiting  these  systems  outright  is  mis¬ 
guided.  Instead,  we  propose  a  three-tiered  process  for  regulating  the  devel¬ 
opment,  deployment  and  use  of  autonomous  systems. 


1.  This  paper  expands  on  arguments  laid  out  in  Kenneth  Anderson  &  Matthew 

Waxman,  Law  and  Ethics  for  Autonomous  Weapon  Systems:  Why  a  Ban  Won't  Work  and  Hou’  the 
Laws  of  War  Can,  Hoover  Institution  (Apr.  9,  2013), 

http:/ / www.hoover.org/  research/law-and-ethics-autonomous-weapon-systems-why-ban- 
wont-work-and-how-laws-war-can;  also  available  as  American  University,  WCL  Research 
Paper  2013-11,  Columbia  Public  Law  Research  Paper  13-351  (2013), 

http:/ / papers.ssrn.com/ sol3/ papers.cfm?abstract_id=2250126. 

2.  U.S.  Department  of  Defense,  DoD  Directive  3000.09,  Autonomy  in  Weapon  Sys¬ 
tems  13  (2012)  [hereinafter  DoD  Directive  3000.09]. 


387 


International  lunv  Studies 


2014 


II.  The  Incremental  Development  of  Autonomous  Weapon 

Systems 


Autonomous  weapon  systems  have  existed  in  some  sense  for  a  long  time. 
Many  anti-personnel  landmines  and  undersea  mines  could  be  thought  of  as 
autonomous  in  that  once  deployed  they  were  configured  (originally  me¬ 
chanically  and  later  electronically)  to  detonate  based  on  physical  contact  or 
other  signals  associated  with  targets.  Many  did  so  in  very  unsophisticated 
ways,  unable  to  distinguish  much  more  than  pressure  or  movement,  let 
alone  select  between  lawful  and  unlawful  targets.  Conceptually,  at  least, 
they  might  fit  the  definition  of  autonomy  if  “select”  is  constmed  to  mean 
merely  “triggered”  rather  than  “selection  among”  targets.  This  paper  fo¬ 
cuses  instead,  however,  on  technologically  sophisticated  systems  in  which 
capabilities  for  “selection  among”  is  a  specific  design  aim  for  the  weapon. 
Some  modern  and  very  sophisticated  autonomous  (or  at  least  very  highly 
automated)  weapon  systems  already  exist.  These  are  generally  limited  to  use 
in  defensive  contexts  against  other  machines  and  are  deployed  in  environ¬ 
ments  such  as  the  air  or  sea  in  which  civilian  risk  is  very  small,  and  with 
respect  to  which  human  operators  activate  and  monitor  the  system  and  can 
override  its  operation. 

New  autonomous  weapon  systems  are  gradually  becoming  incorpo¬ 
rated  into  warfare  as  technology  advances  and  capabilities  increase  one 
small  automated  step  at  a  time.  Increasing  automation  in  weapons  technol¬ 
ogy  results  from  advances  in  sensor  and  analytical  capabilities  and  their  in¬ 
tegration  into — and  in  response  to  the  increasing  tempo  of — military  oper¬ 
ations.  It  also  results  from  political  pressures  to  protect  not  just  one’s  own 
personnel,  but  also  civilian  persons  and  property.  '  Although  automation 
will  be  a  general  feature  across  battlefield  environments  and  weapon  sys¬ 
tems,  genuine  autonomy  in  weapons  will  probably  remain  rare  for  the  fore¬ 
seeable  future  and  be  driven  by  special  factors  such  as  speed  and  the  tempo 
required  of  particular  kinds  of  operations.4 


3.  Kenneth  Anderson  &  Matthew  Waxman,  Op-Ed,  Killer  Robots  and  the  Laws  of  War, 

WALL  Street  Journal,  Nov.  4,  2013,  at  A19,  available  at 

http://onHne.wsj.eom/news/articles/SB10001424052702304655104579163361884479576. 

4.  See  Peter  Singer,  Wired  for  War:  The  Robotics  Revolution  and  Con¬ 

flict  IN  THE  21ST  CENTURY  (2009).  For  a  dissenting  view  that  removing  humans  from 
military  targeting  is  unlikely  for  the  foreseeable  future,  see  Werner  }.  A.  Dahm,  Op-Ed, 
Killer  Drones  Are  Science  Diction,  WALL  STREET  JOURNAL  (Feb.  15,  2012), 

http://onHne.wsj.eom/news/articles/SB1000142405297020488330457722159001547518 

0. 
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“Automation”  describes  a  continuum,  and  there  are  various  ways  to  de¬ 
fine  places  along  it.5 6  Terms  like  “semi-autonomous,”  “man-in-the-loop” 
and  “man-on-the-loop”  are  used  to  convey  different  levels  and  configura¬ 
tions  of  machine  -human  interaction  and  degrees  of  independent  machine 
decision  making.'1  Rather  than  engaging  in  definitional  debates,  our  analyti¬ 
cal  starting  point  is  that  new  autonomous  systems  will  develop  incremen¬ 
tally  as  more  functions,  not  just  of  the  weapon  but  also  of  the  platform 
(e.g.,  the  vehicle  or  aircraft),  are  automated. 

For  example,  intermediate  levels  of  automation  of  weapon  systems 
could  include  a  robot  that  is  pre-programmed  to  look  for  certain  enemy 
weapon  signatures  and  to  alert  a  human  operator  of  the  threat,  who  then 
decides  whether  or  not  to  pull  the  trigger.  At  a  next  level  of  automation, 
the  system  might  be  set  with  the  human  being  not  required  to  give  an  af¬ 
firmative  command,  but  instead  merely  deciding  whether  to  override  and 
veto  a  machine-initiated  attack.  Perhaps  next  the  system  would  be  designed 
to  target  and  fire  autonomously,  but  to  wait  and  call  for  higher-level  au¬ 
thorization  only  when  it  assesses  possible  collateral  damage  above  a  certain 
level.  In  some  cases,  a  human  operator  might  control  only  a  single  or  very 
few  sets  of  sensor  and  weapon  units.  In  others,  he  might  control  or  over¬ 
see  an  integrated  network  of  sensor  and  weapon  units.  In  still  other  cases, 
the  move  to  automate  the  weapon  system  or  even  give  it  autonomy  might 
be  driven  by  automation  of  all  the  other  non-weapon  systems  of  the  plat¬ 
form  with  which  the  weapon  has  to  be  coordinated.  Eventually,  these  sys¬ 
tems  may  reach  the  point  of  full  autonomy  for  which,  once  activated,  the 
human  role  is  vanishingly  small.  As  explained  below,  the  tipping  point 
from  a  highly-automated  system  to  an  “autonomous”  one  is  very  thin  and, 
in  practice,  unstable. 

Weapon  systems  that  would  be  able  to  assess  civilian  status  or  estimate 
harm  as  part  of  their  own  independent  targeting  decisions,  as  in  one  of  the 
above  examples,  do  not  exist  today  and  research  toward  such  capabilities 
currently  remains  in  the  realm  of  theory.  Still,  several  modern  highly- 
automated — and  some  would  call  them  autonomous — weapon  systems 
already  exist.  These  are  generally  for  use  in  battlefield  environments  such 
as  naval  encounters  at  sea  where  risks  to  civilians  are  small,  and  are  general- 


5.  See  Defense  Science  Board,  U.S.  Department  of  Defense,  Task  force  Re¬ 
port:  The  Role  of  Autonomy  in  DoD  Systems  3-5, 23-24  (2012). 

6.  See  William  C.  Marra  &  Sonia  K.  McNeil,  Understanding  “ The  Loop”:  Regulating  the 
Next  Generation  of  War  Machines,  36  HARVARD  j  OURNAL  OF  LAW  AND  PUBLIC  POLICY  1139 
(2013). 
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ly  limited  to  defensive  contexts  against  other  machines  in  which  human 
operators  activate  and  monitor  the  system  and  can  override  its  operation. 
Examples  include  the  U.S.  Patriot  and  Phalanx  anti-missile  systems  and 
Israel’s  Iron  Dome  anti-missile  system.7 8  Many  more  could  lie  ahead  for  a 
variety  of  battlefield  environments  and  military  missions  in  a  future  that  is 
becoming  less  and  less  distant. 

From  a  “demand-side”  perspective,  i.e.,  the  needs  of  the  military  opera¬ 
tor,  increasing  automation  in  weapons  technology  is  an  unsurprising  re¬ 
sponse  to  the  increasing  tempo  of  military  operations  in  which,  other 
things  being  equal,  the  faster  system  wins  the  engagement.  Automation 
permits  military  systems  of  all  kinds  (not  just  weapons)  to  act  more  quickly 
than  people  can  assess,  calculate  and  respond,  and  sometimes  to  act  more 
precisely  and  accurately  in  responding  to  a  military  threat.  Some  of  the  ear¬ 
liest  computational  weapon  systems  emerged  in  World  War  II,  such  as  the 
development  of  anti-aircraft  guns  that  had  rudimentary  mechanisms  for 
estimating  trajectory  and  responding  to  enemy  aircraft,  targeting  the  antici¬ 
pated  path  of  the  enemy  bomber  and  firing  in  what,  for  that  period,  was 
high  speed.9 

Speed  of  response  is  a  demand  of  military  necessity,  certainly,  but  it  is 
not  simply  a  question  of  winning  the  engagement.  Speed  through  automa¬ 
tion  or  autonomy  can  also  serve  to  make  the  use  of  force  in  battle  more 
precise.  By  shortening  the  time  between  the  positive  identification  of  a  tar¬ 
get  and  its  attack,  there  is  less  likelihood  that  the  situation  might  have 
changed,  the  target  have  moved  or  civilians  have  come  into  proximity.  In 
the  Libya  hostilities  in  2011,  for  example,  NATO-manned  attack  aircraft 
were  reportedly  too  slow  and  had  too  little  loiter  time  to  permit  accurate 
targeting  of  highly  mobile  vehicles  on  the  ground  in  an  urban  battlefield 


7.  See  International  Committee  of  the  Red  Cross,  Report  of  the  ICRC  Ex¬ 
pert  Meeting  on  “Autonomous  weapon  systems:  technical,  military,  legal 
AND  HUMANITARIAN  ASPECTS,”  (May  9,  2014),  available  at  http://www.icrc.org/ 
eng/ assets/ files/2014/ expert-meeting-autonomous-weapons-icrc-report-2014-05-09.pdf 
[hereinafter  REPORT  OF  ICRC  EXPERT  MEETING]. 

8.  U.S.  Navy  Fact  File,  MK-1 5  Phalanx  Close-In  W'eapons  System  (CIWS),  http://www. 
navy.mil/navydata/fact_display.asp?cid=2100&tid=487&ct=2  (Nov.  15,  2013);  Federa¬ 
tion  of  American  Scientists,  MK  15  Phalanx  Close-In  Weapons  System  (CIWS),  (Jan.  9,  2003), 
http://www.fas.org/man/dod-101/sys/ship/weaps/mk-15.htm;  Michael  N.  Schmitt  & 
Jeffrey  S.  Thurnher,  Out  of  the  Coop:  Autonomous  Weapon  Systems  and  the  Caiv  of  Armed  Conflict , 
4  Harvard  National  Security  Journal  231  (2013). 

9.  Another  example  is  the  Norden  Bombsight,  a  basic  analog  computer  used  by  the 
U.S.  Air  Force  in  WWII  to  greatly  increase  bombing  accuracy. 


390 


Adapting  the  Caiv  of  Armed  Conflict 


Vol.  90 


with  many  civilians.  In  response,  an  appeal  was  made  to  the  United  States 
to  supply,  first,  surveillance  drones,  and  then  armed  drones  that  could 
speed  up  the  targeting  process.1"  Some  version  of  this  problem  will  drive 
demand  for  automation.  On  top  of  all  this,  there  are  always  powerful  pres¬ 
sures  domestically  and  internationally  on  armed  forces  for  advanced  weap¬ 
on  technology  to  protect  not  just  one’s  own  personnel,  but  also  civilian 
persons  and  property. 

From  a  “supply-side”  perspective,  increasing  automation — and  eventu¬ 
ally  autonomy — of  weapon  systems  grows  from  ever-continuing  advances 
in  sensor  and  analytic  technologies,  machine-learning  and  their  fusion.  Im¬ 
portantly,  development  of  many  of  the  enabling  technologies  of  autono¬ 
mous  weapon  systems — artificial  intelligence  and  robotics,  for  example — 
are  being  driven  by  private  industry  for  many  commercial  and  societally- 
beneficial  purposes  (consider  self-driving  cars,  surgical  robots,  and  so  on). 
They  are  developing  and  proliferating  rapidly,  independent  of  military  de¬ 
mand  and  investment.11  Such  civilian  automated  systems  are  already  mak¬ 
ing  daily  decisions  that  have  potential  life  and  death  consequences,  such  as 
aircraft  landing  systems.  While  most  people  are  generally  aware  that  these 
types  of  systems  are  highly  automated  (or  even  autonomous  for  some 
functions),  and  have  become  wholly  comfortable  with  their  use,  relatively 
little  public  discourse  has  addressed  the  increasing  decision-making  role  of 
autonomous  systems  in  potentially  life-threatening  situations.  To  the  extent 
that  such  automation  and  robotics  technologies  come  to  be  widely  under¬ 
stood  as  more  effective,  safe  and  reliable  than  human  judgment  in  many 
non-military  realms,  their  use  will  almost  certainly  migrate  into  military 
ones.  Indeed,  future  generations  that  perhaps  come  to  routinely  tmst  the 
computerized  judgments  of  self-driving  vehicles  are  likely  to  demand,  as  a 
moral  matter,  that  such  technologies  be  used  to  reduce  the  harms  of  war.  It 
is  largely  a  question  of  whether  such  systems  work  or  not,  and  how  well. 


10.  See,  eg,  Julian  E.  Barnes,  U.S.  Launches  Drone  Strikes  in  Libya,  WALL  STREET  JOUR¬ 
NAL,  Apr.  22,  2011,  at  A6,  available  at  http://online.wsj.com/news/articles 

/SB 1 000 1 4240527 4870488940457 6277413211 029304  (“Allied  aircraft  have  proved  unable 
to  fully  protect  civilians  and  rebel  forces  under  heavy  attack  from  Gadhafi  loyalists.  The 
drones  can  get  closer,  allowing  more-precise  strikes,  and  loiter  longer,  giving  them  a  better 
chance  of  finding  hidden  targets.  .  .  .  Drones  have  been  used  for  reconnaissance  missions 
from  the  start  of  the  conflict,  but  in  recent  days,  NATO  commanders  had  asked  the  U.S. 
to  provide  armed  Predator  strikes.”). 

11  .See  Robert  O.  Work  &  Shawn  Brimley,  20  YY:  Preparing  for  War  in  the 
Robotic  Age  23-27  (2014). 
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No  doubt,  the  use  and  especially  proliferation  of  autonomous  weapons 
will  pose  significant  risks  and  challenges  for  law  and  regulation.  We  address 
below  some  legal  objections  to  autonomous  weapons  as  such.  But  it  bears 
recognizing  at  the  outset  that,  as  with  any  technologically  advanced  weapon 
system,  there  are  risks  and  dangers  that  include  machine  malfunction  and 
machines  whose  design  (hardware,  software  or  interaction  with  human  op¬ 
erators)  underperforms  a  legally  essential  task  or  produces  unpredictable 
effects  (including  when  autonomous  weapon  systems  engage  each  other  or 
not-fully  predictable  machine  decisions  resulting  from  probabilistic  pro¬ 
gramming  or  machine  (self-)  learning).1-  Beyond  issues  of  the  autonomous 
weapon  system  itself,  strategic  or  political  issues  include  concerns  about  a 
State  armed  with  autonomous  weapons  being  too  willing  to  resort  to  mili¬ 
tary  force  because  these  weapons  might  reduce  perceived  risks  to  a  side’s 
own  soldiers  or  to  civilians.13  Moreover,  these  systems  might  be  thought  to 


12.  For  an  introduction  to  robotics  using  machine  self-learning  and  probabilistic  pro¬ 
gramming,  see  Ryan  Calo,  Robotics  and  the  New  Cyberlaw,  103  CALIFORNIA  LAW  REVIEW 
101,  125-26  (forthcoming  2015),  advance  draft  available  at  www.robotics  businessre- 
view.com/pdfs/roboticscyberlaw.pdf  (describing  how  machine  self-learning  might  pro¬ 
duce  machine  behaviors  that  cannot  be  fully  predicted  in  advance).  For  a  technical  intro¬ 
duction  to  the  probabilistic  programming  or  machine  learning  utilized  in,  for  example, 
Google’s  self-driving  cars,  see  SEBASTIAN  THRUN,  WOLFRAM  BURGARD  &  DIETER  FOX, 
Probabilistic  Robotics  (2005).  For  arguments,  contra,  that  autonomous  weapon  systems 
will  never  in  fact  be  able  to  perform  adequately  because  of  limitations  of  the  ability  to 
program  their  behaviors,  see,  e.g.,  Noel  E.  Sharkey,  The  Evitability  of  Autonomous  Robot  War¬ 
fare,  94  International  Review  of  the  Red  Cross  787  (2012),  available  at 
http:/ /www.icrc.org/  eng  /  resources/ documents/ article/ review-2012/irrc-886-sharkey.htm. 
For  arguments  that  as  a  matter  of  fundamental  morality  targeting  and  engagement  deci¬ 
sions  should  not  be  entrusted  to  robots,  and  raising  concerns  about  accountability  for 
mistake,  errors  and  design  flaws,  see,  e.g.,  Peter  Asaro,  On  Tanning  Autonomous  Weapon  Sys¬ 
tems:  Human  Rights,  Automation,  and  the  Dehumanisation  of  Eethal  Decision-Making,  94  INTER¬ 
NATIONAL  review  OF  THE  Red  Cross  687  (2012),  available  at  http://www.icrc.org/ 
eng / resources/documents/article/ review-2012/irrc-886-asaro.htm;  WENDELL  WALLACH 
&  Colin  Allen,  Moral  Machines:  Teaching  Robots  Right  from  Wrong  47-48 
(2010). 

13.  Autonomous  weapon  systems  and  remotely-piloted  armed  drones  raise  the  same 
issue:  whether  the  features  of  the  weapon  system  that  limit  risk  to  soldiers  and  make  it 
more  discriminating  with  respect  to  civilians  are,  ironically,  the  very  features  that  make  it, 
on  some  views,  not  just  easier  for  a  party  to  resort  to  force,  but  “too  easy”  to  do  so.  See, 
e.g.,  UK  Ministry  of  Defence,  JDN  2/11,  The  UK  Approach  to  Unmanned  Air¬ 
craft  Systems  ch.  5  (2011);  Medea  Benjamin,  Drone  Warfare:  Killing  By  Re¬ 
mote  CONTROL  124-43  (rev.  ed.  2013).  For  a  response,  see  Kenneth  Anderson,  Efficiency 
in  Bello  and  ad  Bellum:  Making  the  Use  of  Force  Too  Easy?,  in  TARGETED  KILLINGS:  LAW 
AND  MORALITY'  IN  AN  ASYMMETRICAL  World  374  (Claire  Finkelstein,  J ens  David  Ohlin 
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undermine  individual  disciplinary  and  accountability  systems  in  the  law  of 
armed  conflict.14 

Note  that  many  of  these  of  these  concerns — machine  malfunction, 
marginal  diminution  of  some  political  or  ethical  constraints  on  using  force, 
and  misuse  or  abuse — are  not  unique  to  autonomous  weapons.  They  are 
true  of  many  military  technologies  and  long-standing  targeting  practices, 
including  artillery,  stand-off  manned  aircraft,  missiles,  rockets  and  other 
over-the-horizon  weaponry.  None  of  these  weapon  systems  is  regarded 
today  as  inherently  illegal;  the  question  is  one  of  lawful  use. 

Autonomous  weapons  also  offer  important  potential  benefits  though, 
not  only  in  terms  of  military  effectiveness,  but  in  terms  of  humanitarian 
protection  as  well.  Existing  systems  mentioned  earlier  help  protect  friendly 
forces  and  populations  from  modes  of  attack  that  are  too  fast  or  complex 
for  human  reaction  and  decision  making.  Like  remotely  piloted  vehicles, 
some  autonomous  weapon  systems  can  operate  without  exposing  person¬ 
nel  to  the  direct  risk  of  enemy  fire.  This,  in  turn,  reduces  pressures  on 
combatant  personnel  to  resort  to  greater  force  to  address  threats,  with  the 
possibility  of  greater  harms  resulting  to  civilians  and  civilian  objects. 

Autonomous  weapon  systems  may  also  reduce  risks  to  civilians  by 
making  targeting  more  precise  and  firing  decisions  more  controlled,  espe¬ 
cially  compared  to  human-soldier  failings  that  are  so  often  exacerbated  by 
panic,  vengeance  or  other  emotions,  as  well  as  the  limits  of  human  senses 
and  cognition.  Empathy  and  sympathy  are  among  the  better  angels  of  hu¬ 
man  nature,  but  one  of  Human  Rights  Watch’s  significant  claims  in  Losing 
Humanity — that  a  fundamental  objection  to  autonomous  weapon  systems  is 
that  they  take  these  emotions  out  of  battlefield  targeting  and  firing  deci¬ 
sions — flies  in  the  face  of  how  much  of  the  stmcture  of  the  law  of  armed 
conflict  exists  to  address,  imperfectly,  the  effects  of  human  soldiers’  battle¬ 
field  emotions,  starting  with  fear,  anger  and  vengeance,  exacerbated  under 
conditions  of  hunger,  exposure,  uncertainty  and  so  on.1'1  The  International 
Committee  of  the  Red  Cross  (ICRC)  has  taken  a  sensibly  cautious  ap¬ 
proach  to  this  question,  observing  in  a  2011  report  that  “emotion,  the  loss 


&  Andrew  Altman  eds.,  2012),  avaitabte  as  Efficiency  in  Bello  and  ad  Bellum:  Targeting  Killing 
Through  Drone  Warfare  at  http:/ / papers.ssrn.com/ sol3/ papers. cfm?abstract_id=l  812124. 

14.  See,  e.g..  Human  RIGHTS  WATCH,  LOSING  HUMANITY:  THE  CASE  AGAINST  KILL¬ 
ER  ROBOTS  sec.  VI  (Problems  of  Accountability  for  Fully  Autonomous  Weapons)  (2012), 
available  at  http://www.hrw.org/reports/2012/ll/19/losing-humanity-0  [hereinafter  LOS¬ 
ING  Humanity]. 

15.  Id.  at  37-38. 
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of  colleagues  and  personal  self-interest  is  not  an  issue  for  a  robot,  and  the 
record  of  respect  for  IHL  by  human  soldiers  is  far  from  perfect,  to  say  the 
least.”16  Weapon  systems  with  greater  and  greater  levels  of  automation 
could — at  least  in  some  battlefield  contexts — reduce  misidentification  of 
military  targets,  better  detect  or  calculate  possible  collateral  damage,  or  al¬ 
low  for  using  a  smaller  quanta  of  force  compared  to  human  decision  mak- 

17 

tng. 

That  is  not  to  say  that  autonomous  systems  do  not  raise  some  difficult 
issues.  Perhaps  foremost  among  these  is  the  fact  that  as  machine-learning 
and  artificial  intelligence  technologies  develop,  it  is  becoming  increasingly 
clear  that  human  beings  may  not  necessarily  always  be  able  to  understand 
how  (and  possibly  why)  autonomous  systems  make  decisions.  In  some  cas¬ 
es,  this  is  a  question  of  the  complexity  of  the  system,  making  it  realistically 
impossible  to  fully  predict  in  advance  how  the  system  will  behave,  to  un¬ 
derstand  the  reasoning  behind  the  system’s  decision  making  as  it  happens 
or  even  to  reconstmct  after  the  fact  how  it  did  behave.  In  other  cases,  the 
nature  of  the  programming  is  probabilistic,  making  it  impossible  to  know 
with  complete  certainty  what  the  machine’s  decision  will  be.18  This  realiza¬ 
tion  may  lie  at  the  core  of  many  of  the  fears  and  concerns  that  have  fueled 
the  campaign  to  ban  autonomous  systems,  discussed  further  below.  But  are 
autonomous  weapon  systems  truly  “game  changers,”  with  respect  to  the 
seeming  loss  of  direct  human  control  over  weapons  and  their  use  on  the 
battlefield? 

We  do  not  believe  they  are.  Even  today,  operators  of  highly- automated 
military  equipment,  similar  to  users  of  civilian  technology,  quite  often  have 
very  little  idea  as  to  exactly  how  the  technology  they  are  using  functions 
internally  as  an  electronic  system,  nor  even  the  logical  process  of  how  the 
system  reaches  results,  conclusions  or  decisions.  And  yet,  the  fact  that  we 
do  not  understand  the  internal  mechanics  (or  electronics  or  software  pro- 


16.  International  Committee  of  the  Red  Cross,  International  Humani¬ 
tarian  Law  and  the  Challenges  of  Contemporary  Armed  Conflict:  Report 
Prepared  for  the  31st  International  Conference  of  the  Red  Cross  and  Red 
CRESCENT  40  (2011),  available  at  http:/ / www.icrc.org/ eng/ resources/ documents/ report/ 
31-international-conference-ihl-challenges-report-2011-10-31.htm  [hereinafter  ICRC  31ST 
Conference  Report], 

17.  See  Shane  R.  Reeves  &  William  J.  Johnson,  Autonomous  Weapons:  Are  Yon  Sure  These 
Are  Killer  Robots?  Can  We  Talk  About  It?,  Army'  Lawyer,  Apr.  2014,  at  25-26. 

18.  See  Calo,  supra  note  12,  at  125-26  (discussion  of  “emergent  behavior”  and  ma¬ 
chine  learning  enabling  the  machine  to  self-improve,  but  with  possibly  unpredictable 
learning  paths  and  results). 
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gramming)  of  a  system  does  not  mean  that  we  are  unable  to  use  such  a  sys¬ 
tem  within  our  existing  rule  structures.  It  also  does  not  mean  that  the  hu¬ 
mans  operating  the  equipment  do  not  have  a  general  understanding  of  the 
operating  parameters  of  the  system  and  its  general  capabilities  and  limita¬ 
tions.  As  will  be  discussed  in  further  detail  below,  the  law’s  focus  has  al¬ 
ways  been — and  should  continue  to  be — on  (a)  ensuring  that  a  weapon 
system  is  not  inherently  unlawful  and  (b)  ensuring  that  the  individual  sol¬ 
dier  or  commander  deploying  or  activating  the  lawful  weapon  system  does 
so  in  accordance  with  the  laws  of  warfare,  to  include  ensuring  that  such 
systems  are  not  deployed  in  situations  for  which  they  were  not  designed 
and  which  could  cause  unnecessary  or  unjustified  harm.  This  important 
relationship  between  the  weapon  system  and  its  human  operator  will  mn 
true  for  autonomous  systems  as  well. 

III.  Misguided  Calls  to  Prohibit  Autonomous  Weapons  Per  Se 


Some  critics  argue  that  legal  and  ethical  deficiencies  of  autonomous  weap¬ 
ons — or  the  policy  tendencies  that  autonomous  weapon  capabilities  would 
unleash — demand  that  such  systems  be  prohibited.  The  assumptions  be¬ 
hind  calls  to  ban  autonomous  weapon  systems  and  the  form  of  proposed 
bans  vary. 

Some  doubt  that  technology  can  ever  be  good  enough  to  make  suffi¬ 
ciently  precise  decisions  to  meet  the  legal  and  ethical  requirements  of  dis¬ 
tinction  or  proportionality,  which  are  discussed  below.  1  Others  believe 
that  these  legal  or  ethical  principles  inherently  require  human  judgment — 
that  lethal  targeting  is  or  should  always  be  guided  directly  by  a  moral  agent 
who  can  be  held  accountable  for  culpable  failures.20  And  others 
acknowledge  that  though  autonomous  weapon  systems  may  not  be  unlaw¬ 
ful  under  existing  law,  especially  since  technological  development  is  uncer¬ 
tain  and  what  costs  and  benefits  any  particular  system  might  offer  is  today 
merely  speculative,  they  nevertheless  should  be  banned  (at  least  for  the 
foreseeable  future,  pending  thorough  international  discussion  and  agree- 


19.  See  LOSING  HUMANITY,  supra  note  14,  at  30-35;  Noel  Sharkey,  Op-Ed,  America’s 

Mindless  Killer  Robots  Must  Be  Stopped,  GUARDIAN  (Dec.  3,  2012), 

http:/ / www.theguardian.com/ commentisfree/ 2012/ dec/ 03/ mindless-killer-robots. 

20.  See,  LOSING  HUMANITY,  supra  note  14,  at  35-36;  Asaro,  supra  note  12. 
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ment  as  to  their  proper  legal  framework)  as  a  prophylactic  precautionary 
measure  because  of  risks  they  pose,  both  known  and  unknown."1 

One  way  to  ban — or  to  interpret  existing  law  as  banning — autonomous 
weapons  is  to  define  some  maximum  level  of  autonomy  for  any  weapon 
system  and  prohibit  any  machine  system  that  exceeds  it.  A  variant  ap¬ 
proach  is  to  define  some  minimum  legal  level  of  human  control.  Human 
Rights  Watch,  for  example,  has  called  for  a  preemptive  “ban  [on]  fully  au¬ 
tonomous  weapons,”  which  “should  apply  to  robotic  weapons  that  can 
make  the  choice  to  use  lethal  force  without  human  input  or  supervision.” 
It  also  proposes  to  ban  their  “development,  production,  and  use,”  as  well 
as  calling  for  “reviews”  of  “technologies  and  components  that  could  lead 
to  fully  autonomous  weapons.”22  The  International  Committee  for  Robot 
Arms  Control,  an  organization  dedicated  to  reducing  threats  from  military 
robotics,  calls  for  the  “prohibition  of  the  development,  deployment  and 
use  of  armed  autonomous  unmanned  systems.”23  A  British  non¬ 
governmental  organization  dedicated  to  the  regulation  of  certain  weapons 
argues  that  lethal  decision  making  should  require  “meaningful  human  con¬ 
trol.”24  This  idea  of  requiring  a  minimum  level  of  “meaningful  human  con¬ 
trol”  emerged  as  a  major  theme  in  discussions  among  States  and  advocacy 
groups  at  the  2014  UN  CCW  meeting.25  Instead  of  encouraging  a  perma- 


21.  Special  Rapporteur  on  Extrajudicial,  Summary  or  Arbitrary  Executions,  Report, 
20-21,  U.N.  Human  Rights  Council,  U.N.  Doc.  A/HRC/23/47  (Apr.  9,  2013)  (by  Chris- 
tof  Heyns)  [hereinafter  Report  of  the  Special  Rapporteur] .  It  appears  that  for  some  in  the 
autonomous  weapons  debate,  either  a  ban  or  moratorium  is  a  virtuous  application  of 
some  form  of  the  precautionary  principle — genetically,  a  policy  heuristic  of  “first,  do  no 
harm”  (i.e.,  prefer  little  risk  of  harm  or  reduction  of  risk  even  if  it  means  giving  up  large 
potential  benefits).  See,  e.g.,  Brian  Rappert  et  al.,  The  Roles  of  Civil  Society  in  the  Development  of 
Standards  around  New  Weapons  and  Other  Technologies  of  Warfare,  94  INTERNATIONAL  Review 
OF  THE  Red  CROSS  765,  767  (2012)  (endorsing  “putting  in  place  so-called  precautionary 
measures  .  .  .  such  as  initially  limiting  use”). 

22.  losing  Humanity,  supra  note  14,  at  46-47. 

23.  Original  Mission  Statement,  ICRAC,  http://icrac.net/statements/  (last  visited  July 
29,2014). 

24.  Memorandum  from  Article  36  for  Delegates  to  the  Convention  on  Certain  Con¬ 
ventional  Weapons  (CCW),  Article  36:  Structuring  Debate  on  Autonomous  Weapon  Sys¬ 
tems  (Nov.  2013),  http://www.article36.org/wp-content/uploads/2013/ll/  Autono- 
mous-weapons-memo-for-CCW.pdf. 

25.  See  Chairperson  of  the  Meeting  of  Experts,  UN  Convention  on  Certain  Conven¬ 

tional  Weapons,  Report  of  the  2014  Informal  Meeting  of  Experts  on  Lethal  Autonomous 
Weapons  Systems  (LAWS)  TJ  20,  http://www.unog.ch/80256EDD006B8954/% 

28%20httpAssets%29/350D9ABEDlAFA515C1257CF30047A8C7/$file/Report_Advan 
cedVersion_10June.pdf  (last  visited  July  29,  2014). 
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nent  ban  on  autonomous  weapons,  Christof  Heyns,  the  UN  Special  Rap¬ 
porteur  on  Extrajudicial,  Summary  or  Arbitrary  Executions,  has  proposed  a 
moratorium,  calling  for  “all  States  to  declare  and  implement  national  mora- 
toria  on  the  testing,  production,  assembly,  transfer,  acquisition,  deployment 
and  use  of  [lethal  autonomous  robotics]  until  such  time  as  an  international¬ 
ly  agreed  upon  framework  .  .  .  has  been  established.”26 

Before  addressing  some  of  the  enforceability  problems  and  dangers  of 
any  effort  to  prohibit  autonomous  weapons,  it  is  important  to  understand 
the  proposed  formulas  do  not,  as  it  may  seem  initially,  contain  a  bright  line 
that  would  be  useful  for  promoting  adherence.  Lawyers  experienced  in  the 
law  of  armed  conflict  will  quickly  see  that  each  of  these  seemingly  clear-cut 
definitions  leaves  many  open  questions  as  to  what  systems  would  be 
banned  under  any  particular  formulation.  Even  something  as  seemingly 
plain  as  “lethal  decision  making”  by  a  machine  does  not  address,  among 
other  things,  the  lawfulness  of  targeting  a  tank,  ship  or  aircraft  which  is 
ultimately  the  source  of  the  threat,  but  inside  of  which  is  a  human  combat¬ 
ant. 

Beyond  definitions,  the  technology  and  basic  architecture  of  an  auton¬ 
omous  system  and  a  nearly  autonomous,  highly  automated  system  are  basi¬ 
cally  the  same — if  you  can  build  a  system  that  is  nearly  autonomous,  for 
example  with  human  override,  then  you  can  probably  reprogram  it  to  elim¬ 
inate  that  human  role.  Moreover,  whether  a  highly  automated  system — say, 
one  with  a  human  supervisor  who  can  override  proposed  firing  deci¬ 
sions — is  in  practice  operating  autonomously  depends  on  how  it  is  being 
manned,  how  operators  are  trained  and  how  effectively  oversight  is  exer¬ 
cised.  It  also  depends  on  operational  context  and  conditions,  which  may 
limit  the  degree  to  which  the  human  role  is  in  any  way  meaningful. 

For  these  and  other  reasons,  a  fully  autonomous  system  and  a  merely 
highly-automated  system  will  be  virtually  indistinguishable  to  an  observer 
without  knowing  a  lot  about  how  that  system  is  used  in  particular  opera¬ 
tional  conditions.  The  difference  might  not  matter  very  much  in  practice, 
given  the  variable  performance  of  human  operators.  In  any  case,  these  sys¬ 
tems  will  be  easily  transitioned  from  one  to  the  other.  The  blurriness  of 
these  lines  means  that  it  will  be  very  difficult  to  draw  and  enforce  prohibi¬ 
tions  on  “fully”  autonomous  systems  or  mandates  for  minimum  levels  of 
human  decision  making.  Given  the  great  practical  difficulty  of  distinguish¬ 
ing  between  autonomous  and  highly  automated  systems,  applying  a  legal 


26.  Report  of  the  Special  Rapporteur,  supra  note  21,  at  21. 
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ban  on  autonomous  systems  would  be  relatively  easy  to  circumvent  and 
very  difficult  to  enforce. 

At  the  same  time,  and  as  alluded  to  above,  imposing  a  general  ban  on 
autonomous  systems  could  carry  some  highly  unfavorable  consequences — 
and  possibly  dangers.  These  could  include  providing  a  clear  advantage  in 
autonomous  weapon  technology  to  those  States  which  generally  would  not 
join  (or  in  reality  comply  with)  such  a  ban.  They  could  also  include  losing 
out  on  the  numerous  potential  advantages  of  autonomous  systems  of  im¬ 
proving  decision  making  on  the  battlefield,  including  through  avoiding 
emotion-based  response;  improving  system  accuracy,  thereby  probably 
minimizing  collateral  injuries;  and  possibly  limiting  human  loss  of  life  on 
both  sides  and  among  civilians. 

IV.  Regulating  the  Use  of  Autonomous  Weapons  under  the 
Law  of  Armed  Conflict 


Calls  for  a  prohibitory  ban  are  usually  based  on  creating  new  treaty  law. 
Without  a  new  treaty,  however,  any  new  weapon  system  must  still  comply 
in  its  design  and  usage  with  the  law  of  armed  conflict.  Article  36  of  Proto¬ 
col  I,  for  example,  requires  that  “in  the  study,  development,  acquisition  or 
adoption  of  a  new  weapon,  means  or  method  of  warfare,  a  High  Contract¬ 
ing  Party  is  under  an  obligation  to  determine  whether  its  employment 
would,  in  some  or  all  circumstances,  be  prohibited  by  this  Protocol  or  by 
any  other  rule  of  international  law  applicable  to  the  High  Contracting  Par¬ 
ty.”27  Additionally,  assuming  a  weapon  is  not  per  se  illegal,  its  operational 
uses  must  also  comply  with  the  law  of  targeting. 


27.  Protocol  Additional  to  the  Geneva  Conventions  of  12  August  1949,  and  Relating 
to  the  Protection  of  Victims  of  International  Armed  Conflicts  art.  36,  June  8,  1977,  1125 
U.N.T.S.  3  [hereinafter  API].  Article  36  is  widely  regarded  among  scholars  as  expressing 
customary  law  with  respect  to  “legal  reviews  of  new  means  of  warfare  before  their  use  is 
generally  considered,”  but  such  consensus  is  “lacking  as  to  whether  an  analogous  require¬ 
ment  exists  to  perform  legal  reviews  of  new  methods  of  warfare.”  Means  of  warfare  refers 
to  “weapons  and  weapon  systems,”  whereas  methods  refer  to  the  “tactics,  techniques,  and 
procedures”  by  which  hostilities  are  conducted.  See  Schmitt  &  Thurnher,  supra  note  8,  at 
271.  Given  that  neither  the  United  States  nor  Israel  is  a  party  to  Protocol  I,  and  given  that 
the  authors  include  former  national  security  government  officials  of  each,  we  believe  it 
prudent  to  observe  that  neither  State  appears  to  have  expressed  an  official  legal  view  as  to 
the  customary  law  status  of  each  aspect  of  Article  36;  however,  the  military  establishments 
of  both  States  have  long  maintained  highly-developed  and  formalized  processes  for  re¬ 
viewing  the  legality  of  weapons.  The  processes  are  similar  to  what  Article  36  requires  of 
State  parties  to  Protocol  I. 
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The  legality  of  the  weapon  itself  turns  on  three  basic  rules.28  First,  the 
weapon  system  cannot  be  indiscriminate  by  nature.29  This  is  not  a  question 
of  whether  there  might  be  circumstances — the  manner  of  use  and  battle¬ 
field  conditions — in  which  the  weapon  could  not  be  aimed  in  a  way  to 
comply  with  the  legal  requirement  of  “distinction”  between  lawful  military 
targets  and  civilians.  That  would  be  tme  of  nearly  any  weapon,  because  any 
weapon  could  be  deliberately  misused.  Rather,  the  rule  mns  to  the  nature 
of  the  weapon  in  the  uses  for  which  it  was  designed 311  or,  as  some  authorities  have 
put  it,  its  “normal”  uses,  i.e.,  the  uses  for  which  it  was  intended.  As  a  mat- 


28.  See,  e.g..  Judge  Advocate  General,  U.S.  Air  Force,  AFI51-402,  Legal  Reviews  of 
Weapons  and  Cyber  Capabilities  3.1.1,  3.1.2  (2011)  (Technical  legal  guidance  for  the  legal 
review  of  weapon  systems,  specifying  the  scope  of  legal  review  of  weapons  to  encompass 
both  specific  treaty  rules  and  customary  law  rules  regarding  the  use  of  the  weapon,  and 
beyond  that  to  consider,  even  in  the  absence  of  an  express  prohibition,  whether  the  weap¬ 
on  is  “of  a  nature”  to  inflict  superfluous  injury  or  unnecessary  suffering  upon  combatants 
and  whether  it  is  “capable  of  being  directed  against  a  specific  military  objective  and,  if  not, 
is  “of  a  nature  to  cause  an  effect  on  military  objectives  and  civilians  or  civilian  objects 
without  distinction.”). 

29.  We  leave  aside  here  a  number  of  technical  complications.  For  example,  notwith¬ 
standing  the  use  of  “of  a  nature”  by  the  U.S.  Air  Force  weapons  review  guidance  with 
respect  to  indiscriminate  weapons,  the  language  “of  a  nature”  appears  in  Article  35(2)  of 
the  Additional  Protocol  I  to  the  Geneva  Conventions,  with  reference  not  to  indiscriminate 
weapons  with  respect  to  civilians,  but  only  weapons  of  a  nature  to  cause  superfluous  inju¬ 
ry  or  unnecessary  suffering  to  combatants.  See  API,  supra  note  27,  art.  35(2).  There  is  rea¬ 
sonably  widespread  State  practice  and  agreement  that  a  similar  concept  applies  to  indis¬ 
criminate  weapons,  with,  however,  considerable  debate  over  the  question  of  how  this  “na¬ 
ture”  is  to  be  determined — whether  by  reference  to  a  weapon’s  design,  “normal”  purpose 
and  use,  etc.  See  William  FI.  Boothby,  Weapons  and  THE  Law  OF  Armed  Conflict 
69-85  (2009).  In  general,  we  have  followed  the  approach  taken  by  both  Boothby,  and 
Schmitt  and  Thurnher. 

30.  It  bears  noting  that  even  the  ban  on  blinding  laser  weapons — the  leading  example 
of  a  ban  on  a  technologically  advanced  weapon  that  had  not  yet  been  deployed — is  limited 
to  “laser  weapons  specifically  designed,  as  their  sole  combat  function  or  as  one  of  their 
combat  functions,  to  cause  permanent  blindness  to  unenhanced  vision.”  Protocol  on 
Blinding  Laser  Weapons  art.  1,  Oct.  13,  1995,  1380  U.N.T.S.  370.  It  underscores  the  gen¬ 
eral  rule  that  the  lawfulness  of  a  weapon  as  such  turns  on  the  use  for  which  it  was  de¬ 
signed  or  intended. 

31.  Yoram  Dinstein,  The  Conduct  of  Hostilities  Under  the  Law  of  Armed 
CONFLICT  62  (2d  ed.  2010)  (“It  is  necessary  to  differentiate  between  (a)  weapons  that  are 
employed  in  specific  circumstances  contrary  to  the  principle  of  distinction  .  .  .  and  (b) 
weapons  that  by  their  very  nature  or  design  cannot  possibly  maintain  the  distinction  in  any 
set  of  circumstances.  The  fact  that  certain  weapons  are  used  indiscriminately  in  a  particu¬ 
lar  military  engagement  does  not  stain  the  weapons  themselves  with  an  indelible  imprint 
of  illegality  . .  . .”). 
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ter  of  grounds  to  find  a  weapon  unlawful,  this  is  a  tough  standard  and  very 
few  weapons  are  illegal  per  se  because  they  are  indiscriminate  by  nature. 
Rather,  legal  weapons  are  used  in  an  indiscriminate  manner — a  serious  vio¬ 
lation  of  the  law  of  armed  conflict,  certainly,  but  running  to  the  actual  use 
of  the  weapon.3" 

Second,  a  lawful  weapon  system  cannot  be  “of  a  nature”  to  cause  “un¬ 
necessary  suffering  or  superfluous  injury.”  ’1  This  provision  aims  to  protect 
combatants  from  needless  or  inhumane  suffering,  such  as  shells  filled  with 
glass  shards  that  would  not  be  detectable  by  an  x-ray  of  the  wound. 14  It  is  a 
mle  that  applies  solely  to  combatants,  not  civilians  (who  are  protected  by 
other  law  of  armed  conflict  provisions).35  Like  the  “indiscriminate  by  na¬ 
ture”  rule,  it  sets  a  high  bar;  unsurprising,  given  the  forms  of  violence  that 
can  lawfully  be  inflicted  upon  combatants  in  armed  conflict. 

Third,  a  weapon  system  can  be  deemed  illegal  per  se  if  the  harmful  ef¬ 
fects  of  the  weapon  are  not  capable  of  being  “controlled.”36  The  mle 
against  weapons  with  uncontrollable  harmful  effects  is  paradigmatically 
biological  weapons,  in  which  a  virus  or  other  biological  agent  cannot  be 
controlled  or  contained;  once  released,  it  goes  where  it  goes.  Once  again, 
even  though  many  rules  of  the  law  of  armed  conflict  prevent  the  use  of 
weapons  in  circumstances  that  might  have  uncontrolled  effects,  the  bar  to 
make  the  weapon  itself  illegal  per  se  is  high. 

None  of  these  rules  renders  a  weapon  system  illegal  per  se  solely  on  ac¬ 
count  of  it  being  autonomous.  If  a  fully  autonomous  weapon  system  were 
supplied  with  sufficiently  reliable  parameters  and  it  were  able  to  act  on 
them  so  as  to  be  able  to  strike  specific  targets  on  the  same  legal  terms  of 


32.  Schmitt  &  Thumher,  supra  note  8,  at  276  (weapons  reviews  “examine  only  the  le¬ 
gality  of  a  weapon  system  as  such,  not  its  use  in  any  particular  circumstance.”). 

33.  API,  supra  note  27,  art.  35(2). 

34.  For  more  examples,  see  COMMENTARY  ON  THE  ADDITIONAL  PROTOCOLS  OF  8 
June  1977  to  the  Geneva  Conventions  of  12  August  1949, 1 1419  (Yves  Sandoz, 
Christophe  Swinarski  &  Bruno  Zimmermann  eds.,  1987). 

35.  Schmitt  &  Thumher,  supra  note  8,  at  244. 

36.  API,  sipra  note  27,  art.  51  (4) (c);  Schmitt  &  Thumher,  supra  note  8,  at  250  (“[Arti¬ 
cle  51  (4)  (c)]  reflects  customary  international  law.  It  disallows  weapon  systems  that,  despite 
being  able  to  strike  their  targets  accurately,  have  uncontrollable  effects.”).  Because  the  rule 
concerns  “effects,”  the  claim  that  some  might  make  is  that  autonomous  weapon  systems 
are  “uncontrollable”  because  for  a  weapon  system  equipped  with  sophisticated  probabilis¬ 
tic  programming  not  every  decision  taken  by  the  machine  would  be  predictable  in  ad¬ 
vance,  thus  would  be  by  definition  uncontrolled.  But  apart  from  other  aspects  of  control 
of  the  machine,  uses  and  operations,  the  rule  is  about  effects  that  cannot  be  uncontrolled, 
not  an  uncontrolled  weapon. 
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discrimination  that  would  apply  to  a  human  soldier,  that  the  weapon  sys¬ 
tem  was  “autonomous”  would  not  violate  the  “indiscriminate  by  nature” 
mle.  Although  some  might  view  an  autonomous  weapon  system  as  “un¬ 
controllable,”  its  effects  are  not  uncontrollable  within  the  meaning  of  the 
legal  provision.  ’7 

Even  if  a  weapon  system  is  not  per  se  illegal,  it  might  still  be  prohibited 
in  some — even  many — battlefield  environments,  or  in  particular  uses  on  a 
particular  battlefield.  But  in  other  circumstances,  of  course,  the  weapon 
might  also  be  legal.  As  the  ICRC  put  the  point  in  2011  with  respect  to  new 
weapon  technologies  generally,  the  question  is  not  whether  the  “new  tech¬ 
nologies  are  good  or  bad  in  themselves,  but  instead  what  are  the  circum¬ 
stances  for  their  use.”38 

Targeting  law  governs  the  circumstances  of  the  use  of  lawful  weapons 
and  includes  three  fundamental  mles:  discrimination  (or  distinction),  pro¬ 
portionality  and  precautions  in  attack.  ’9  Distinction  requires  that  a  combat¬ 
ant,  using  reasonable  judgment  in  the  circumstances,  distinguish  between 
combatants  and  civilians,  as  well  as  between  military  and  civilian  objects.4" 
Although  use  of  autonomous  weapon  systems  is  not  illegal  per  se,  a  re¬ 
quirement  for  their  lawful  use — the  ability  to  distinguish  lawful  from  un¬ 
lawful  targets — might  vary  enormously  from  one  weapon  system’s  tech¬ 
nology  to  another.  Some  algorithms,  sensors  or  analytic  capabilities  might 
perform  well,  others  poorly. 

Moreover,  these  capabilities  are  measured  against  particular  battlefield 
environments;  the  “context  and  environment  in  which  the  weapon  system 
operates  play  a  significant  role  in  this  analysis.”41  Air-to-air  combat  between 
military  aircraft  over  the  open  ocean,  for  example,  might  one  day  take  place 
between  autonomous  systems,  as  the  technological  pressures  for  greater 


37.  Schmitt  and  Thumher  are  right  to  conclude  that  the  “likelihood  of  an  autono¬ 
mous  weapon  system  being  unlawful  per  se  is  very  low”  on  any  of  the  three  grounds  de¬ 
scribed  above.  Schmitt  &  Thumher,  supra  note  8,  at  250. 

38.  ICRC  31st  Conference  Report,  supra  note  16,  at  40. 

39.  See  Ian  Henderson,  The  Contemporary  Law  of  Targeting:  Military 
Objectives,  Proportionality,  and  Precautions  under  Additional  Protocol  I 
23-41  (2009). 

40.  API,  sipra  note  27,  art.  48  (“Parties  to  the  conflict  shall  at  all  times  distinguish  be¬ 
tween  the  civilian  population  and  combatants  and  between  civilian  objects  and  military 
objectives  . .  . .”). 

41.  Jeffrey  S.  Thumher,  The  Tali'  that  Applies  to  Autonomous  Weapon  Systems ,  17  AMERI¬ 
CAN  Society  of  International  Law  Insights  (Jan.  18,  2013), 

www.asil.org/insights/ volume /17/issue/ 4/law-applies-autonomous-weapon-systems. 


401 


International  lunv  Studies 


2014 


speed,  ability  to  endure  torque  and  inertial  pressures,  and  so  on,  finally  re¬ 
sult  in  an  unmanned  craft.  As  the  speed  of  flight  and  every  other  aircraft 
system  becomes  faster  to  the  point  that  only  an  autonomous  system  can  fly 
the  plane,  at  that  point  and  at  those  speeds  the  weapon  systems  may  also 
have  to  be  automated  in  order  to  operate  effectively  with  the  rest  of  the 
aircraft  systems. 

Distinction  is  highly  unlikely  to  be  an  issue  in  that  particular  operation¬ 
al  environment,  however,  because  the  combat  environment  would  be  lack¬ 
ing  in  civilians.  As  Jeffrey  Thurnher  points  out,  there  may  be  “situations  in 
which  an  autonomous  weapon  system  could  satisfy  this  rule  with  a  consid¬ 
erably  low  level  ability  to  distinguish  between  civilian  and  military  tar¬ 
gets.”42  Yet  there  would  be  many  operational  environments  in  which  meet¬ 
ing  the  requirements  of  distinction  by  a  fully  autonomous  system  would  be 
very  difficult — urban  battlefield  environments  in  which  civilians  and  com¬ 
batants  are  commingled,  for  example.  This  is  not  to  say  that  autonomous 
systems  are  thereby  illegal.  Quite  the  opposite,  in  some  settings  their  use 
would  be  legal  and  in  others  illegal,  depending  on  how  technologies  ad¬ 
vance. 

Proportionality  requires  that  the  reasonably  anticipated  military  ad¬ 
vantage  of  an  operation  be  weighed  against  the  reasonably  anticipated  civil¬ 
ian  harms.43  As  with  the  principle  of  distinction,  there  are  operational  set¬ 
tings — air-to-air  combat  over  open  water,  tank  warfare  in  remote  uninhab¬ 
ited  deserts,  ship  antimissile  defense,  undersea  anti-submarine  operations, 
for  example — in  which  civilians  are  not  likely  to  be  present  and  which,  in 
practical  terms,  do  not  require  weighing  military  advantage  against  civilian 
harms.  Conversely,  in  settings  such  as  urban  warfare,  proportionality  is 
likely  to  pose  very  difficult  conditions  for  machine  programming,  and  it  is 
widely  recognized  that  whether  and  how  such  systems  might  one  day  be 
developed  is  simply  an  open  question.44  To  be  fair,  many  military  lawyers 
have  questioned  whether  human  soldiers  are  capable  of  truly  applying  this 
ambiguous  test  either. 

Programming  proportionality  is  not  simply  a  question  of  sensors  or 
computational  sophistication  in  order  to  reasonably  identify  a  person  as  a 
lawful  target  in  the  sense  of  distinction.  While  everyone  agrees  that  civilian 
harm  should  not  be  excessive  in  relation  to  military  advantages  gained,  the 


42.  Id. 

43.  The  customary  law  rule  is  codified  in  Additional  Protocol  I.  API,  supra  note  27, 
arts.  51(5) (b),  57(2)(a)(iii). 

44.  See  ICRC  31  ST  Conference  Report,  supra  note  16. 
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comparison  is  apples  and  oranges.  Although  there  is  a  general  sense  that 
such  excess  can  be  determined  in  truly  gross  cases,  there  is  no  accepted 
formula  that  gives  determinate  outcomes  in  specific  cases. 4:1  Some  military 
lawyers  proceed  largely  casuistically,  building  on  what  was  done  in  prior 
situations  and  examining  similarities  and  differences.46  Difficult  or  not, 
proportionality  is  a  fundamental  requirement  of  the  law  and  any  completely 
autonomous  weapon  system  would  have  to  be  able  to  address  it,  though,  as 
with  distinction,  reasonable  judgments  of  proportionality  would  be  highly 
dependent  on  the  operational  environment  and  battlefield  in  which  the 
machine  was  deployed.47  Again,  assessing  proportionality  is  one  thing  in 
close-in  infantry  urban  warfare,  but  altogether  different  in  remote  uninhab¬ 
ited  deserts,  in  the  open  ocean  or  undersea,  or  circumstances  of  machine- 
on-machine  operations  where  few  if  any  civilians  are  present. 

Precautions  in  attack  require  that  an  attacking  party  take  feasible  pre¬ 
cautions  in  the  circumstances  to  spare  the  civilian  population.46  Precautions 
and  feasibility,  it  bears  stressing,  are  terms  of  art  in  the  law  of  armed  con¬ 
flict  that  confer  reasonable  discretion  on  commanders  undertaking  at- 


45.  See  William  H.  BOOTHBY,  The  Law  of  Targeting  96-97  (2012)  (doctrinal  dis¬ 
cussion  of  the  difficulties  of  comparisons  in  specific  cases  versus  general  principles). 

46.  As  Schmitt  notes,  human  targeting  officers  in  U.S.  forces  sometimes  employ  so¬ 

phisticated  software  programming  to  estimate  likely  collateral  damage  from  the  employ¬ 
ment  of  a  particular  munition  in  particular  circumstances,  given  assumptions  about  a  vari¬ 
ety  of  factors.  See  Michael  N.  Schmitt,  Autonomous  Weapon  Systems  and  IHL:  A  Reply  to  the 
Critics,  Harvard  National  Security'  Journal  19  (Feb.  15,  2013), 

http:/ /harvardnsj.org/2013/02/ autonomous-weapon-systems-and-international-humanit 
arian-law-a-reply-to-the-critics/.  These  software  tools  are  merely  tools,  dependent  on  the 
assumptions  used  and  only  address  the  likely  collateral  damage — not  the  comparison  to 
military  advantage.  See  also  Geoffrey  S.  Corn  et  al.,  The  Law  of  Armed  Conflict: 
An  Operational  Approach  194  (2012)  (“One  process  for  implementing  ROE  that  is 
currently  in  use  in  Afghanistan  by  U.S.  forces  is  the  Collateral  Damage  Estimation  Meth¬ 
odology  or  CDEM.  .  .  .  |T]he  purpose  of  the  CDEM  is  to  cause  the  military  to  do  a  thor¬ 
ough  proportionality  analysis  and  then,  depending  on  the  estimated  number  of  civilian 
casualties,  seek  approval  for  the  attack.”). 

47.  Thurnher,  supra  note  41  (“[F]or  the  use  of  an  autonomous  weapon  system  to  be 
lawful,  the  system  would  be  expected  to  reasonably  distinguish  between  combatants  and 
civilians  .  .  .  given  the  particular  environment  and  circumstances  of  the  battlefield  ruling  at 
the  time.”). 

48.  API,  supra  note  27,  arts.  57,  58;  for  explanation,  see,  e.g.,  DlNSTElN,  supra  note  31, 
at  138-40;  HENDERSON,  supra  note  39,  at  157-96. 
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tacks.4’  The  commander’s  obligation  is  grounded  in  reasonableness  and 
good  faith,  and  in  “planning,  deciding  upon  or  executing  attacks,  the  deci¬ 
sion  taken  by  the  person  responsible  has  to  be  judged  on  the  basis  of  all 
information  available  to  him  at  the  relevant  time,  and  not  on  the  basis  of 
hindsight.”5"  Although  some  precautionary  measures  might  have  to  be  pro¬ 
grammed  into  a  genuinely  and  fully  autonomous  weapon  system,  in  reality 
many  of  the  precautions  that  might  be  at  issue  are  not  so  much  a  question 
of  what  the  weapon  system  does  in  a  particular  missile  strike  as  part  of  a 
larger  operation  or  battle,  but  instead  are  addressed  in  the  planning  for  the 
overall  attack  as  an  operation,  including  the  development  of  the  rules  of 
engagement.  As  such,  and  for  the  foreseeable  future,  the  rules  of  precau¬ 
tions  in  attack  fall  upon  commanders  as  they  plan  how  to  deploy  their 
combat  resources  in  an  operation.  Technology  can  be  of  great  assistance  in 
modeling  likely  harm  to  civilians  or  civilian  objects  or  in  deciding  upon  the 
munitions  or  weapon  systems  (including  at  some  future  point,  perhaps, 
autonomous  systems)  to  employ,  but  responsibility  for  the  operation,  in¬ 
cluding  the  weapons  used,  belongs  to  the  commander.51 

While  it  is  tme  that  for  this  and  many  other  legal  reasons  a  fully  auton¬ 
omous  weapon  system  would  likely  have  to  be  able  to  break  off  or  alter  an 
engagement — potentially  even  as  it  was  being  launched  (in  at  least  some 
circumstances,  where  “feasible”) — just  as  a  human  commander  might  be 
required  to  do,  precautions  in  attack  operate  generally  at  a  level  above  an 
individual  weapon,  whether  operated  by  a  human  soldier  or  by  a  computer 
integrated  into  it,  and  rely  on  pre-planning  and  information  obtained  be¬ 
forehand.  Situational  awareness  for  those  directly  carrying  out  the  attack  is 
often  very  low,  even  if  the  target  is  visible  to  them,  and,  because  they  do 
not  typically  have  awareness  of  the  entire  operation  and  its  parts,  they  are 
often  not  in  a  position  to  make  a  judgment  as  to  a  change  in  precautions, 
and  so  cannot  be  held  to  account  for  a  violation  of  reasonably  planned 
precautions.  The  legal  standard,  in  principle,  would  be  no  different  for  a 


49  We  note  that  while  the  United  States  and  Israeli  armed  forces  exercise  strict  precautions 
in  attack,  their  position  on  the  customary  legal  status  of  such  obligation  is  not  publicly 
formalized. 

50.  See  W.  Hays  Parks,  The  Protection  of  Civilians  from  Air  Warfare ,  27  ISRAEL  YEAR¬ 
BOOK  ON  HUMAN  Rights  65,  110  (1997)  (quoting  German  declaration  on  Article  57, 
Protocol  I). 

51.  See  Gregory  S.  McNeal,  Targeted  Killing  and  Accountability,  102  GEORGETOWN  LAW 
JOURNAL  681,  739-50  (2014)  (discussing  the  collateral  damage  estimation  and  mitigation 
process,  and  evaluating  precautions  in  attack  through  weapon  selection  and  use  of  weap¬ 
ons  in  targeting  and  targeted  killing  operations). 
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machine  carrying  out  only  one  part  of  a  larger  operation,  though  policy 
might  lead  to  greater  caution.  Precautions  in  attack,  with  respect  to  new 
weapon  technologies,  in  the  long  run  might  turn  out  to  be  the  most  diffi¬ 
cult  requirement  of  targeting  law  for  an  autonomous  system  to  meet,  at 
least  in  circumstances  in  which  there  are  civilians  present,1"  precisely  on 
account  of  the  difficulties  of  anticipating  and  pre-planning  conditions  and, 
if  the  machine  were  to  be  genuinely  on  its  own,  identifying  and  executing 
attacks.  But  independence  for  such  a  machine  system,  divorced  from  the 
integration  of  all  levels  of  planning  for  an  attack  and  its  execution,  is  frank¬ 
ly  a  long  way  away,  if  not  entirely  fanciful. 

Many  of  the  battlefield  environments  for  which  autonomous  weapon 
systems  are  contemplated  today  do  not  involve  such  considerations,  how¬ 
ever;  they  are  not  situations  with  significant  civilian  presence.  Still,  con¬ 
sistent  with  other  aspects  of  the  interplay  of  automation  technology  and 
the  law  of  armed  conflict,  incremental  changes  in  technology  will  push  for 
use  of  these  systems  in  more  and  varied  environments  in  which  civilians 
are  indeed  present.  Legal  reviews  will  likely  have  to  take  into  account  many 
particularized  facts — all  the  permutations  of  in-  and  on-the-loop  interac¬ 
tions  by  soldiers  with  the  machine,  that,  for  example,  is  still  short  of  full 
autonomy,  but  nonetheless  has  very  large  roles  for  automation. 

Stepping  back,  it  is  critically  important  to  understand  that  before  an  au¬ 
tonomous  weapon  system — like  any  weapon  system — is  used  in  a  military 
operation,  human  commanders  and  those  employing  the  weapon  will  gen¬ 
erally  continue  to  be  expected  to  exercise  judgment  about  the  likely  pres¬ 
ence  of  civilians  and  the  likelihood  that  they  may  be  inadvertently  harmed; 
expected  military  advantage;  particular  environmental  features  or  condi¬ 
tions;  the  weapon’s  capabilities,  limitations,  and  safety  features;  and  many 
other  factors.  It  is  difficult  to  draw  general  conclusions  in  the  abstract 
about  the  many  complex  legal  issues  involved  in  such  scenarios.  However, 
in  many  cases,  even  though  a  weapon  system  may  be  autonomous,  much 
of  the  required  legal  analysis  would  be  conducted  by  human  decision  mak¬ 
ers  who  elect  whether  or  not  to  use  it  in  a  specific  situation.  Whether  legal 
requirements  are  satisfied  in  a  given  situation  will  still  depend  not  simply 
on  the  machine’s  own  programming  and  technical  capabilities,  but  on  hu¬ 
man  judgments  as  well.  In  thinking  through  these  legal  issues,  it  may  be 
helpful  also  to  distinguish  between  human  decisions  to  deploy  a  system 


52.  See  William  H.  Boothby,  Some  Legal  Challenges  Posed  by  Remote  Attack,  94  INTERNA¬ 
TIONAL  Review  of  the  Red  Cross  579  (2012). 
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(including  decisions  about  settings  for  the  system)  in  a  particular  circum¬ 
stance  and  decisions  made  by  the  system  itself  once  deployed.  These  two 
sets  of  decisions  are  linked  in  complex  ways,  and  making  the  former  deci¬ 
sions  appropriately  should  reduce  the  possibility  for  error  in  the  latter. 

In  sum  there  is  no  reason,  in  principle,  why  a  highly  automated  or  au¬ 
tonomous  system  could  not  satisfy  the  requirements  of  targeting  law.3  ’  Like 
any  otherwise  lawful  weapon,  it  depends  on  the  use  and  the  environment. 
Uninhabited  deserts  are  different  from  urban  warfare.  Destroying  rockets 
in  the  air — Iron  Dome,  for  example — is  different  from  mral  counterinsur¬ 
gency  in  which  entering  civilian  villages  is  necessary,  not  just  for  bare  secu¬ 
rity  reasons,  but  to  consolidate  political  relationships.  In  practical  terms, 
autonomous  systems  might  be  better  able  to  satisfy  the  law  in  some  uses 
and  environments  than  others,  but  that  is  not  a  matter  of  principle;  it  is  a 
matter  of  whether  and  how  far  technological  capability  advances  relative  to 
the  legal  standard. 

V.  Developing  and  Cultivating  Legal  Rules  and  Codes  of 

Conduct 

The  conclusions  that  autonomous  weapon  systems  do  not  inherently  vio¬ 
late  the  longstanding  and  accepted  rules  of  warfare,  and  that  such  systems 
can  be  effectively  regulated  under  these  rules,  does  not  mean  that  this  will 
be  a  simple  or  straightforward  exercise.  The  advent  of  autonomous  sys¬ 
tems  creates  new  challenges  that  need  to  be  addressed. 

It  is  quite  rare  for  an  international  law-related  question  to  arise  be¬ 
fore  it  actually  becomes  a  real-life  dilemma.54  There  is  therefore  a  unique 
(although  probably  short-lived)  opportunity  to  get  it  right;  to  develop  the 
mles  and  code  of  conduct  for  such  systems  before  they  are  fielded  on  the 
battlefields  of  the  world  in  large  numbers.  Though  primarily  procedural, 
the  best  way  to  approach  this  and  to  effectively  adapt  the  law  of  armed 
conflict  to  future  autonomous  weapons  is  a  three-tiered  approach. 

At  the  highest  level,  some  form  of  international  instmment  addressing 
autonomous  systems  is  probably  needed  eventually.  But  the  instrument 
that  is  needed  is  neither  a  blanket  ban  on  autonomous  weapons  nor  a  mor¬ 
atorium  on  their  development  and  deployment.  At  its  most  basic,  such  an 


53.  Schmitt  &  Thumher,  supra  note  8,  at  279-80. 

54.  Compare,  for  example,  issues  of  autonomous  weapon  systems  to  cyber  warfare  is¬ 
sues,  where  the  law  is  likewise  struggling  to  keep  up  with  technological  and  real-life  devel¬ 
opments. 
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instrument  would  make  clear  that  autonomous  systems  are  governed  by  the 
existing  rules  of  warfare,  that  they  are  not  inherently  illegal  thereunder  and 
that  they  are  subject  to  the  customary  law  requirements  of  legal  review  of 
weapons. 

Such  an  instmment  could  go  much  further,  however,  elaborating  on 
two  separate  sets  of  substantive  issues.  First,  it  could  include  some  form  of 
interpretative  application  of  the  law  of  armed  conflict  to  commanders  de¬ 
ploying  autonomous  systems,  explaining  what  information  such  command¬ 
ers  must  have  and  what  questions  such  commanders  must  ask  before  de¬ 
ciding  to  field  the  weapons  in  a  given  situation.  Second,  it  could  include 
mles  and  guidelines  for  the  development  of  autonomous  systems.  Such 
mles  and  guidelines  could  be  based  not  only  on  legal  requirements,  but  also 
on  policy  considerations.  For  example,  do  we  want  to  require  all  autono¬ 
mous  weapon  systems  to  have  a  human  override  capability  (some  form  of 
“kill  switch”)?  Or  perhaps  we  would  like  all  such  systems  to  have  a  built-in 
self-neutralization  mechanism.  Perhaps  this  instrument  could  include  a 
minimum  sensory  requirement  standard  for  autonomous  systems  that 
could  be  required  to  make  reasonably-foreseeable  distinction-based  deci¬ 
sions  on  the  battlefield? 

This  international  instmment  need  not  deal  with  all  of  these  issues  at 
once.  Indeed,  as  explained  below,  a  better  approach  is  to  reach  consensus 
on  some  core  minimum  standards,  but  at  the  same  time  to  retain  some 
flexibility  for  international  standards  and  requirements  to  evolve  as  tech¬ 
nology  evolves.  Such  an  instmment  is  not  likely  to  have  compliance  trac¬ 
tion  with  States  over  time  unless  it  largely  codifies  standards,  practices,  pro¬ 
tocols  and  interpretations  that  States  have  converged  upon  over  a  period  of 
actual  development  of  systems.  The  process  of  this  convergence  will  best 
be  accomplished  if  it  takes  place  gradually  through  informal  discussions 
among  States,  informed  by  sufficiently  transparent  and  open  sharing  of 
relevant  information,  rather  than  through  formal  treaty  negotiations  that  if 
initiated  too  early  tend  to  lock  states  into  rigid  political  positions. 

Such  an  international  instmment  could  have  many  different  forms.  It 
could  be  a  totally  new  legal  instrument;  it  could  be  a  new  protocol  under 
the  CCW  or  it  could  be  a  more  manual-based  approach,  similar  to  the  re¬ 
cent  Tallinn  Manual  on  the  International  Taw  Applicable  to  Cjber  Warfare ,  but 
involving  more  direct  participation  and  endorsement  by  States.  Each  ap- 


55.  Tallinn  Manual  on  the  International  Law  Applicable  to  Cyber  War¬ 
fare  (Michael  N.  Schmitt  ed.,  2013). 
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proach  has  its  own  advantages  and  disadvantages,  but  a  variant  on  the  Tal¬ 
linn  Manual  approach  may  be  especially  appropriate  here.  That  document 
was  developed  by  an  international  group  of  legal  experts  commissioned  by 
the  NATO  Cooperative  Cyber  Defence  Centre  of  Excellence  to  develop 
and  propose  interpretive  guidance  for  States’  and  other  actors’  considera¬ 
tion.  Although  the  cyber  context  is  different,  insofar  as  there  may  be  great¬ 
er  disagreement  as  to  the  appropriate  legal  framework,  similar  international 
processes — whether  involving  State  representatives  or  independent  experts 
or  both — can  help  foster  broad  consensus  or  surface  disagreements  that 
require  resolution  with  respect  to  autonomous  weapon  systems. 

The  second  tier  will  be  at  the  national  level.  All  countries,  and  especial¬ 
ly  countries  in  which  autonomous  system  development  is  currently  ongo¬ 
ing,  should  promulgate  their  own  national  mles  and  policies  addressing  the 
two  sets  of  substantive  issues  mentioned  above.  Undoubtedly  many  of  the 
details  of  national  rules  and  policies  will  need  to  remain  secret,  as  they  will 
involve  sensitive  matters  of  military  capabilities  and  practices.  States  should 
be  urged,  however,  to  publish  openly  their  general  policies  and  to  promote 
sharing  of  best  practices.  A  step  in  this  direction  was  the  U.S.  Defense  De¬ 
partment’s  development  and  publication  of  a  policy  directive  regulating 
automated  and  autonomous  weapon  systems  that  spells  out  limits  and  pro¬ 
cedural  requirements  with  regard  to  research,  development  and  deploy- 
ment  ot  autonomous  weapon  systems. 

Finally,  the  third  tier  of  a  regulatory  process  should  comprise  the  de¬ 
velopers  and  potential  users:  the  defense  industry  (including  private  sector 
developers  of  component  technologies)  and  responsible  militaries  of  the 
world.  The  former  will  need  clear  guidance  as  to  what  types  of  systems 
they  should  and  should  not  be  developing.  The  latter  need  to  start  thinking 
about  methodologies,  operating  procedures,  rules  of  engagement  and  other 
operational  and  doctrinal  level  rules  for  the  use  of  autonomous  systems. 
While  both  of  these  groups  will  be  generally  implementing  standards  and 
mles  that  flow  down  from  the  higher  tiers,  it  is  crucial  for  them  to  be  in¬ 
volved  in  the  discussions  at  the  higher  levels,  as  in  many  cases  only  they 
will  have  a  clear  understanding  of  the  real  technological  and  operational 
questions  and  dilemmas  that  arise.  For  example,  collaboration  between  pri¬ 
vate  sector  weapon  developers  and  the  militaries  that  might  acquire  and 
deploy  them  will  be  critical  in  formulating  and  implementing  effective  test- 


56.  DoD  Directive  3000.09,  supra  note  2. 
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ing  and  evaluation  systems  specifically  for  autonomous  weapons.5'  It  is 
preferable  that  all  three  tiers  advance  in  parallel,  each  drawing  upon  the 
other  in  the  complicated  dialogue  involving  international  and  national  in¬ 
terests  that  generally  characterizes  the  development  of  international  law. 

Lawyers  knowledgeable  in  the  requirements  of  weapons  legal  reviews 
thus  need  to  be  engaged  with  weapon  designers  and  engineers  in  the  pri¬ 
vate  sector  from  the  beginning  of  the  development  process  in  order  to  help 
ensure  that  weapons  law  be  as  much  a  part  of  the  system’s  design  require¬ 
ments  as  any  other.  Similarly,  they  must  be  engaged  with  military  operators 
able  to  assist  them  in  understanding  the  intended  uses  and  battlefield  envi¬ 
ronments  for  the  system,  as  well  as  the  capacities  and  limits  of  soldiers  us¬ 
ing  the  system  and  requirements  for  training  of  the  operators.  The  devel¬ 
opment — not  just  the  deployment  and  use — of  a  weapon  system  that  is 
autonomous  (or  has  an  autonomous  firing  option)  requires  an  “under¬ 
standing  of  the  legal  parameters;  the  engineering  design,  production,  and 
testing  (or  validation)  methods;  and  the  way  in  which  the  weapon  might  be 
employed  on  the  battlefield.”38  While  a  review  conducted  during  the  acqui¬ 
sition  process  is  not  the  final  legal  review,  the  stages  of  design,  demonstra¬ 
tion,  manufacture  and  in-service  deployment  are  “important  stages  for  the 
input  of  formal  legal  advice.”59  To  be  meaningful,  though,  these  legal  pa¬ 
rameters  must  translate  into  terms  of  reliability  engineering  that  are  “testa¬ 
ble,  quantifiable,  measurable,  and  reasonable.”60 

Military  lawyers  are  well  aware  of  these  issues  and  a  robust  debate  over 
the  legal  review  of  increasingly  automated  weapons  has  been  underway  in 
internal  defense  ministry  publications  and  journals  dedicated  to  the  law  of 
armed  conflict  for  well  over  a  decade.  One  task  is  coordination  and  inte- 


57.  See  Report  of  ICRC  Expert  Meeting,  supra  note  7  (“[I]t  is  not  clear  how  such 
weapons  could  be  adequately  tested  given  the  absence  of  standard  methods  for  testing  and 
evaluating  autonomous  systems.”).  Backstrom  and  Henderson  acknowledge  the  difficul¬ 
ties  in  testing  systems,  but  add  that  “one  pro-active  step  that  could  be  taken  as  part  of  the 
legal  review  process  is  for  lawyers  to  [provide]  input  [into]  the  test  and  evaluation  phases 
by  identifying  areas  of  legal  concern  that  could  then  be  translated  into  testable  elements.” 
Alan  Backstrom  &  Ian  Henderson,  New  Capabilities  in  W-'arfare:  An  Overview  of  Contemporary 
Technological  Developments  and  the  Associated  Legal  and  engineering  Issues  in  Article  36  Weapons 
Reviews,  94  International  Review  of  the  Red  Cross,  483,  507  (2012). 

58.  Backstrom  &  Henderson,  supra  note  57,  at  401. 

59.  Justin  McClelland,  The  Revien’  of  Weapons  in  Accordance  ivith  Article  36  of  Additional 
Protocol  I,  85  International  Review  of  the  Red  Cross  397, 401  (2003). 

60.  Defense  science  Board,  U.S.  Department  of  Defense,  Report  of  the 
Defense  Science  Board  Task  Force  on  Developmental  Test  &  Evaluation  38 
(2008),  available  at  www.acq.osd.mil/ dsb/ reports/ ADA482504.pdf. 
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gration  of  review  tasks  across  disciplines  and  between  government  defense 
agencies  and  the  private  sector,  and  to  address  questions  about  how  to 
translate  legal  requirements  into  concrete  engineering  tests.  A  further  coor¬ 
dination  task  is  creating  dialogue  and  cooperation  among  States  and  mili¬ 
tary  establishments  so  that  internally  generated  legal  standards  and  re¬ 
quirements  and  engineering  tests  help  develop  informal  norms,  expecta¬ 
tions,  standards  and  shared  understandings  that  might,  at  a  suitable  future 
point,  crystallize  into  concrete  national  and/or  international  treaty  mles 
with  respect  to  highly  automated  or  autonomous  weapons. 

The  double  challenge  here  is,  on  the  one  hand,  for  States  to  apply  suf¬ 
ficiently  clear  and  robust  standards  and  rules,  under  reasonable  understand¬ 
ings  of  their  Article  36  obligations,  as  sophisticated,  modern  autonomous 
weapon  systems  are  gradually  fielded.  And,  on  the  other  hand,  ensure  that 
standards  and  rules  under  Article  36  that  States  develop  today  will  be 
equally  relevant  or  adaptable  for  the  future  systems  which  will  be  devel¬ 
oped  ten,  twenty,  thirty  years  from  now. 

The  fundamental  principle  underlying  the  gradual  development  of 
these  standards  and  mles  alongside  the  evolution  of  automation  technolo¬ 
gies,  however,  should  be  that  what  matters  is  ever  greater  compliance  with 
the  core  obligations  of  the  law  of  armed  conflict:  necessity,  distinction, 
proportionality  and  humanity.  Whether  the  actor  on  the  battlefield  is  a 
“who”  or  a  “what”  is  not  truly  the  issue,  but  rather  how  well  that  actor  per¬ 
forms  according  to  the  law  of  armed  conflict.  Debate  over  standards  or 
mles  for  automated  or  autonomous  systems  should  remain  scrupulously 
neutral  as  between  human  or  machine,  and  should  affirmatively  reject  any  a 
priori  preference  for  human  over  machine.  Even  seemingly  indisputable 
calls  for  a  first  principle  of  “meaningful  human  control”  mistake  the  issue, 
which  is  lessening  the  harms  of  armed  conflict  within  the  law  by  the  means 
that  are  the  most  effective.  The  principle  of  humanity  is  fundamental,  but  it 
refers,  not  to  some  idea  that  humans  must  operate  weapons,  but  instead  to 
the  promotion  of  means  or  methods  of  warfare  that  best  protect  humanity 
within  the  lawful  bounds  of  war,  irrespective  of  whether  the  means  to  that 
end  is  human  or  machine  or  some  combination  of  the  two. 

Moreover,  as  previously  suggested,  the  solutions  to  the  challenges 
posed  by  autonomous  weapons  likely  lie  in  a  process  of  gradual  interna¬ 
tional  legal  development  that  evolves  as  technology  advances.  It  is  not  one 
that  seeks  to  predict  how  technology  will  or  will  not  evolve,  or  one  that 
tries  to  pause  technological  development  until  a  new  legal  consensus  is  in 
place.  Hitting  the  pause  button  on  technology,  pending  consensus  on  the 
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legal  framework  to  govern  it,  seems  attractive  for  some  of  those  who  have 
concerns  about  the  risks  of  such  technologies.  But  there  already  is  a  general 
framework  in  place  for  these  emerging  weapons — the  law  of  armed  con¬ 
flict  and  its  fundamental  principles,  and  the  processes  for  the  legal  review 
of  weapons. 


VI.  Conclusion 

As  increasingly  automated — and  in  some  cases  fully  autonomous — weapon 
systems  enter  the  battlefield  or  become  possible,  it  is  important  that  inter¬ 
national  norms  to  regulate  them  head  down  a  path  that  is  coherent  and 
practical.  Contrary  to  the  claims  of  some  advocates,  autonomous  weapon 
systems  are  not  inherently  illegal  or  unethical.  The  technologies  involved 
potentially  hold  considerable  promise  for  making  armed  conflict  more  dis¬ 
criminating  and  causing  less  harm  on  the  battlefield.  They  do  pose  great 
challenges,  however,  with  regard  to  law  of  armed  conflict  mles  regulating 
the  use  of  weapons.  Those  challenges  demand  international  attention  and 
special  processes  for  adapting  existing  law  to  meet  those  challenges. 

Rather  than  seeking  to  impose,  up  front,  a  new  set  of  prohibitory  rules 
or  seeking  to  suspend  development  of  autonomous  weapon  systems  pend¬ 
ing  a  comprehensive  agreement  on  rules  to  govern  them,  international  reg¬ 
ulation  of  autonomous  weapon  systems  should  begin  with  the  premise  that 
the  law  of  armed  conflict  provides  an  appropriate  general  framework. 
States  should  work  to  build  on  that  framework  through  continually- 
improving  interpretive  standards  and  agreed-upon  best  practices.  There  is 
no  risk-free  course  of  action,  and  the  three-tiered  approach  we  propose 
offers  a  way  to  appropriately  and  realistically  constrain  military  activities, 
while  improving  adherence  to  core  law  of  armed  conflict  principles. 
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